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ABSTRACT 

One of the goals of the Strategies and Models for Improved 
Learning through Effective Data Use (SMILE) project was to develop a data- 
capacity survey (DCS) to measure a school's capacity to use data. In this 
study, a DCS was developed and field tested. The DCS, which was field tested 
at six Milwaukee Public Schools (Wisconsin), was a pen-and-paper survey 
designed to be completed as a group activity. The first section included a 
general description, instructions, a glossary, and three different data-use 
scenarios of school improvement, classroom, and project settings. The second 
section of the DCS contained 28 items with rubric-format or checklist format 
response areas. The DCS was designed to measure the concept of data capacity 
by deconstructing it into the inquiry process and sociotechnical dimensions. 
In general, field test results reveal both between- and within-school 
variation along both data capacity dimensions and across the three data-use 
scenarios. Findings further suggest that schools distinguish between 
differences in data capacity for each stage of the inquiry process, and that 
schools distinguish between the social and technical factors. Schools also 
showed variation across the three data-use scenarios. Some design issues 
related to the DCS were identified. The DCS took too much time to complete, 
was too complex, and schools tended to adjust their scores. These design 
(features will be considered in the redesign of the DCS. (Contains 5 figures 
and 10 references.) (SLD) 
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Introduction 

There has been a recent push for schools and districts to make better use of data in 
order to detect and react to problems, document successful interventions and strategies 
(Love, 2002; Schmoker, 1999, 1999b; Bernhardt, 1999; Johnson, 1997), and better 
understand how educational systems work (Willms, 1992). However, the process 
required to use data effectively has numerous inherent costs, many of which are 
encountered well before any benefits can be enjoyed. Schools that are effective data-users 
will minimize those costs where possible, while maximizing the positive effects of data- 
use. Helping schools to become more proficient data-users (i.e., improving their data 
capacity) requires not only an understanding of effective data practices, but also of the 
conditions that need to exist at a school to support those practices. 

From the literature on systemic reform, we see an argument that posits that it is 
not sufficient for schools to want to reform; they must also have the capacity to reform 
(Massell, 1998; O’Day et al., 1995). However, the concept of reform capacity is not well 
understood and is often difficult to define (Massell, 1998). Capacity can only be 
measured in relation to something, such as a well-defined process or event. O’Day et al. 
(1995) define reform capacity as “the ability of the educational system to help all students 
meet more challenging standards” (p. 1). The authors note that reform capacity can be 
thought of as being a multi-dimensional trait. They suggest that teacher capacity is 
comprised of knowledge, skills, dispositions, and views of self. Furthermore, they 
suggest that teacher capacity interacts with organizational capacity (O’Day, 1995), which 
is also multidimensional, including vision and leadership, collective commitment and 
culture, knowledge (or access to knowledge), organizational structures, and resources. 
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Thus, they have systematically deconstructed the concept of reform capacity into more 
measurable traits. 

Extending Massell’s theoretical argument, we argue that schools must not only 
have the desire to use data, but they must also have the capacity to use data. More 
specifically, schools must have the capacity to use data to improve student learning. We 
will refer this as data capacity. One goal of the Strategies and Models for Improved 
Learning through Effective Data Use (SMILE) project is to develop a data-capacity 
survey (DCS) that can measure a school’s data capacity. 

The DCS is designed to serve two purposes. First, schools should be able to use 
the DCS to self-assess their own ability to use data while also receiving feedback 
regarding how to improve their data capacity. Repeated assessments with the DCS 
should provide schools with a way to evaluate the effectiveness of their efforts to 
improve their data capacity. Additionally, researchers should be able to use the 
instrument to study the impact of specific interventions, or otherwise further research in 
how data-use can be linked to improving schools. 

Similar to O’Day’s deconstructionist approach to studying reform capacity 
(O’Day et al., 1995), we have decomposed data capacity into two dimensions - the 
inquiry-process dimension and the sociotechnical dimension. The inquiry-process 
dimension represents the data inquiry process associated with using data within a school 
improvement context. This process can be cyclical and generally follows the steps of 
inquiry, data acquisition, data management, analysis, dissemination, and evaluation 
(Bernhardt, 1999; Love, 2002; Willms, 1992). While each stage is most likely necessary 
to successful data-use, none alone are sufficient. Furthermore, since each step depends on 
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successful completion of the previous steps, problems in early stages of the data inquiry 
process may propagate problems in later stages. 

The sociotechnical dimension differentiates between social and technical factors. 
Support for such a dimension arises from the sociotechnical systems literature, which 
argues that there are social and technical factors that must be addressed when planning 
system design or improvement (Clegg, 2000; Pasmore, 1988). A central axiom, joint 
optimization, from sociotechnical systems theory posits that both social and technical 
factors must be addressed before system performance can be optimized. Examples of 
social factors that are likely to influence data capacity include staff support, 
organizational structures, communication, and organizational culture, whereas examples 
of technical factors include effective use of technology (e.g., district information 
resources or electronic data files), as well other technical processes, such as analyzing 
data and developing internal monitoring systems. 

The remainder of this paper describes our development of the DCS, what we have 
learned from field-testing, and how those findings will impact the redesign of the DCS. 
Specifically, in the methods section we describe how the DCS is designed and how it was 
field-tested. In the results and conclusions section, we focus on analyzing the results of 
the field tests as they pertain to the design of the DCS, with emphasis on identifying the 
strengths, weaknesses, and design issues. Finally, in the discussion section we summarize 
changes to the DCS that are likely to occur as we address our preliminary findings. 
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Methods 

Participants 

The DCS, which was field tested on six Milwaukee Public Schools (MPS), 
comprised by two elementary schools (Kindergarten - 5* grade), two middle schools (6*'' 
- 8'*' grade), and two K-8 schools (Kindergarten - 8'*' grade). The schools vary in size and 
performance and reflect Milwaukee’s cultural diversity. 

Data-Capacitv Survey (DCS) 

The DCS has been developed over the last year, and has incorporated feedback 
from testing sessions in an on-going process. This section will describe the DCS as it 
currently exists, but the reader should be aware that the instrument has evolved 
throughout the testing sessions. 

The DCS is a pen-and-paper survey designed to be completed as a group activity. 
The first section of the survey includes a general description, instructions, glossary of 
terms, and description of three different data-use scenarios - a school-improvement 
scenario, a classroom scenario, and a project scenario. The school improvement scenario 
portrays a situation where a school uses data within the context of developing, writing, 
and substantiating a school improvement plan. The classroom scenario presents an 
example of a school trying to use data to inform classroom changes within the context of 
improving student learning. The project scenario presents an example of a school using 
data to answer or track a school-wide question or program. 
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The second section of the DCS contains a total of 28 items and their response 
formats. Each item is comprised of a stem question and either a rubric-format or 
checklist-formatted response area (Figure 1). There are 18 rubric-formatted questions, for 



Figure 1. Sample item from the DCS with a rubric response format 



12) Which best describes how your school stores and maintains data? 



Check the statement that best 
describes your school. 


Education 

Plan 

Scenario 


Assessing 

Classroom 

Learning 


School 

Wide 

Problem 


Mostly paper 








Some paper, some electronically 








Mostly electronically, no local 
database 








Mostly electronically, at least 
some in local database 









which each school team selects a descriptor (out of four options) that best fits their 
school. Ten checklist format questions (figure 2) require the school team to select any 
and all options that apply to their school. Items are grouped by response format to 
minimize errors caused by changes of survey-response format. Each item also has a 
response field for each of the three data-use scenarios. The rational for having multiple 
scenarios is that the way in which schools use data can vary greatly. Therefore, it is likely 
that a school’s data capacity will also vary according to the context of the data-use. 
During our initial testing, we were curious as to whether or not the context of data use 
would affect schools responses, and if so, how. 
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The DCS is designed to measure the concept of data capacity by deconstructing it 
into the inquiry-process and sociotechnical dimensions. Four to six items assess each 
stage of the data-inquiry process. The first stage, decision-making, refers to the thinking 
and planning that should occur before schools begin to work with data. The data- 
acquisition stage focuses on tasks related to gaining access to and acquiring data. Work 
associated with organizing and maintaining data falls under the data-management stage 



Figure 2. Sample item from the DCS with a checklist response format 

22) Who are likely to use data to reflect and evaluate school and classroom 
decisions? 



Check all that apply 
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and tasks related to making sense of data and sharing conclusions are included together in 
the analyzing and reporting stage. The next stage, application, involves applying 
information and knowledge to improving school or classroom practices. Finally, the last 
stage occurs when schools evaluate their work in order to better understand whether or 
not the changes were successful and why. 

The DCS is also organized on a sociotechnical dimension. This dimension can be 
broken down into social and technical factors. Thirteen items address social factors (see 
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figure 2 for an example), such as how many persons are involved in using data, and 
whether or not sufficient support is provided to those staff. Ten items assess technical 
aspects of data-use (see figure 1 for an example), such as how technology is used, what 
types of analysis methods are used, and other technical processes. Five items do not 
differentiate between social and technical factors. 

Procedure 

Each SMILE school selected was nominated by district leadership and the 
school’s willingness to participate (measured primarily through interest by the school’s 
administration). Schools assembled a team of four to ten staff members that included 
school administration (principal participation was required), teachers, and staff. SMILE 
researchers met with each team during a four-hour planning meeting, during which they 
completed a version of the DCS and established some preliminary project goals. Of this 
four-hour meeting, two hours were scheduled for the DCS; however, school teams 
required at least two and a half hours to complete the survey. 

A SMILE researcher facilitated the completion of the DCS; school teams were 
told what to expect and that they Should answer each item based on how the school is 
currently using data. The facilitator also emphasized that the teams answer each items 
honestly without regard to whether they were being judged or evaluated. In addition, 
teams were instructed to ask questions as they arise. 

School teams were then asked to discuss each of the three data-use scenarios so 
that they could build a shared understanding of the different contexts in which data are 
used. Once they felt they understood the scenarios and what was expected of them, teams 
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were instructed to discuss each item until they had a shared understanding of the item’s 
stem question and had formed a consensus on how to score their school for that item, for 
each data-use scenario. They would then progress to the next item. After completing the 
DCS, teams were given a short break and then completed a series of exercises with the 
purpose of selecting a project focus - while also mapping out factors that are likely to 
impact student learning within that focus. Feedback surveys were administered at the end 
of the four-hour session. 

In addition to completing the DCS, school teams participated in monthly meetings 
with our SMILE staff, which was prepared to assist them in improving their data 
capacity. These monthly meetings produced a significant body of materials, which will 
be analyzed using qualitative methods. The results from the qualitative analysis will also 
be used as a basis for revising the DCS. Schools will then complete the revised DCS in 
September of 2003. 



Results and Conclusions 

Results will be discussed in terms of what the DCS field tests reveal about the 
conception and design of the survey. In general, results reveal between- and within- 
school variation along both data capacity dimensions and across the three data-use 
scenarios. Averaging responses along the inquiry-process dimension yields results that 
suggest that schools distinguish between differences in data capacity for each stage of the 
inquiry-process. For example, one school scored relatively high on the first three stages 
of the inquiry process, but scored relatively low on later stages (figure 3). Note that 
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figure 3 includes a stage for selecting data that has subsequently been eliminated from the 
survey design. 

Figure 3. Analysis of one school along inquiry-process dimension 



School A: Inquiry-Process Dimension 




Inquiry Process Task 



Analyzing responses along the sociotechnical dimension (figure 4) also shows 
that schools can differentiate between the social and technical factors. Figure four shows 
how one school appears organized to use data for a school improvement scenario, but less 
so for using data within the classroom, or to answer a school wide project. However, a 
problem arises when one tries to identify best practices for certain organizational 
structures. For example, a school with a highly centralized data process may have a 
higher capacity than a school using a distributed model, especially for data-use scenarios 
that do not necessarily require distributed participation. This challenge reflects one 
difficulty of creating the DCS, namely that there may be multiple models for effective 
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Figure 4. Analysis of one school across the sociotechnical dimension 



School B: Data-Use Scenario by Capacity Type 
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Schools also showed variation across the three data-use scenarios, providing 
evidence that the context of data-use is an important variable to control for when studying 
how schools use data. In general, schools appeared to score themselves differently for 
each of the three scenarios (figure 5). Looking at interactions between data-use scenario 
and the other two dimensions also suggests that schools scored themselves differently 
depending on the context of using data (figure 4). These trends suggest that schools score 
themselves relatively high on some data-use scenarios and low on others. Thus, the 
context of data-use appears to be an important factor when measuring a school’s data 
capacity. 

The structure of the rubrics appear to be appropriate in that most responses were 
in the bottom half of the four-point scale, suggesting that the DCS response options are 
properly phrased and can potentially show growth in a school’s data-capacity over time. 
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However, this trend is also evidence of a possible floor effect, such as might be the case 
when scores stay low, even in schools that are thought to be schools with high data 
capacity. Future changes will be evaluated in terms of whether or not responses appear 
to be abnormally low or high, and whether or not they show change. 

Figure 5. Analysis of one school across Data-Use Dimension 

School B: Data-Use Scenario Dimension 
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Looking at how the schools completed the DCS reveals several design issues 
related to the administration of the survey. First, the DCS requires too much time to 
complete. None of the schools completed the instrument within the targeted two-hour 
provided for. Furthermore, despite the efforts of the facilitators to keep the teams on 
pace, many were rushed to complete it within 2.5 hours. Understandably, many of the 
school teams showed signs of fatigue towards the end of the DCS. This finding was true 
for all schools, despite on-going efforts to revise the DCS between site visits. This is a 
central design issue that must be addressed. 
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Another problem was that schools were susceptible to adjusting their scores (in 
both directions). Some schools would purposively round their scores down when the 
group was split between two different rubric options so that they could have some room 
to improve. Another school team scored its school relatively high; but based on our 
experiences with this school over the subsequent school year, it would appear that they 
are one of the schools with relatively lower data-capacity. It is unclear what motivated 
this school to inflate its scores. It is also important to note that schools were given 
explicit instructions, along with reminders throughout the session, to answer honestly and 
without regard to how the school might appear, suggesting that these instructions were 
discounted by the team. This is another central issue for the DCS because score inflation 
will impact the validity and reliability of the survey. 

Several school team members reported back to SMILE staff that they valued the 
discussion that arose during their DCS sessions. Indeed, we observed a great deal of 
productive and focused dialogue during the DCS sessions. For example, members would 
often discuss various aspects of how their school uses (or not) a particular source of data. 
Since data-related processes are often distributed across multiple staff members, most 
team members had only a limited understanding of all the data work that occurred at their 
schools. These types of discussion are one reason why the instrument required so much 
time to complete. It is unclear whether shortening the DCS without also eliminating much 
of this data-dialogue is possible. 

Another finding is that schools perceived the DCS as being too complex. One 
source for this perception lies in how the DCS is structured. Namely, teams would first 
read an item’s stem question, discuss its meaning, read and discuss the item’s response 
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options, and then respond to the item for each of the three data-use scenarios. While the 
teams showed some ability to speed up as they progressed through the survey, it was 
nevertheless a slow and deliberate process. Another source for this perception might be 
that the process of using data in schools is often complex, as well as distributed. For 
example, a school may rely on several people within the school to process and use data 
yet those staff members maybe unaware of how their individual responsibilities are 
integrated into an overall system. Therefore, the DCS may be perceived as being too 
complex, not because it is overly structured or too detailed, but because the process of 
using data is more complex than staff had previously understood it to be. This 
explanation is supported by our observations that many team members discovered new 
aspects of how their school uses data and for what purposes. 

Discussion 

In interpreting these findings, it is important to keep in mind that the goals of the 
data capacity survey are two-fold. First, the DCS is designed as a self-assessment tool for 
schools wanting to improve their ability to use data. Furthermore, the DCS is designed to 
be useful to researchers by providing a way to measure a school’s capacity to use data. It 
is unclear at this point whether or not these two goals are compatible. Preliminary results 
suggest that we should continue to use the inquiry-process and sociotechnical dimensions 
as both appear to be useful ways of organizing and understanding the actual processes 
that schools use, as well as the conditions that are necessary to support those processes. 
The question is: How do we make the DCS easier to complete, while at the same time 
improving its utility. 
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One likely method for increasing the utility of the DCS is to develop a guide to 
aide schools in interpreting their results. This guide would help schools self-analyze their 
results to understand what the DCS is telling them about their ability to use data. 
Additionally, in the case where capacity is low, the interpretive guide will suggest where 
and how a school might want to focus their resources. 

A major problem is that the DCS requires too much time to complete. Not 
unrelated to this problem is the perception that the DCS is too complex. An area that is 
likely to be revised is the number of data-use scenarios. Reducing the number of 
scenarios will likely accomplish two goals: First, schools will be able to complete the 
DCS in less time. Second, it will appear to be less complex. The argument against doing 
so is that the validity of the DCS’s may suffer. Furthermore, the rich dialogue observed 
may as well be diminished. While, this eliminating one or two of the data-use scenarios 
will surely reduce the amount of time required to complete the DCS, it will also likely 
decrease the utility of the survey. Another option under review is to split the DCS into 
three versions, one for each data-use scenario. Schools could then complete the DCS 
scenario that best fits their needs - or if they prefer to do all three scenarios, split the total 
time into more manageable sessions. One option not being considered is that of reducing 
the number of items. This option would not necessarily result in significant reductions in 
time. 

A second problem is that the instrument must be redesigned to elicit more reliable 
and valid responses. While it is unclear why one school appeared to inflate its scores, the 
instrument should be reviewed from the perspective of whether or not each item provides 
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a school with concrete and well defined criteria. Finally, improving the saliency of 
instructions that remind schools to respond honestly might be helpful. 

In summary, we plan to separate the DCS into self-contained modules that each 
contain only one of the three data-use scenarios. In addition, a guide will be developed 
for each module to aide schools in interpreting their responses and in providing 
suggestions for where a school might want to focus their resources when trying to 
improve their data capacity. Results from qualitative analysis of field notes and other 
project materials will also be taken into account during the redesign of the DCS. 
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